Image processing device, method of controlling image processing device and program causing computer to execute method

ABSTRACT

There is provided an image processing device including an image acquisition part acquiring an image; a depth acquisition part acquiring a depth associated with a pixel in the image; a depth conversion part converting the depth in accordance with a function having a characteristic to nonlinearly approach a predetermined value with an increase in the depth; and a storage part storing the converted depth in association with the image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/425,563, filed Feb. 6, 2017, which is a continuation of U.S.application Ser. No. 13/549,954, filed on Jul. 16, 2012, now U.S. Pat.No. 9,609,308, issued Mar. 28, 2017, which claims the benefit ofpriority under 35 U.S.C. § 119 from Japanese Application No.2011-182518, filed in the Japan Patent Office on Aug. 24, 2011, theentire contents of all of which are incorporated herein by reference.

BACKGROUND

The present technology relates to an image processing device, a methodof controlling an image processing device and a program causing acomputer to execute the method, and particularly to an image processingdevice, a method of controlling an image processing device and a programcausing a computer to execute the method performing image processingbased on a depth.

In recent years, an image pickup apparatus capable of measuring a depthin association with a pixel in an image becomes popular. An imageprocessing device in an image pickup apparatus can perform imageprocessing such as blur processing (i.e., smoothing processing) forproducing bokeh by using the depth.

For example, an image pickup apparatus that measures depths to subjectsand performs, with focusing on a principal subject, smoothing processingon a background of the principal subject to a degree corresponding to adepth based on the principal subject is disclosed (e.g., Japanese PatentLaid-Open No. 2003-37767). Such smoothing processing is often performedon a background of a portrait photo, for example, for highlighting theperson. Further, because an image pickup apparatus equipped with animage sensor with a small light receiving surface area captures an imagewith relatively small bokeh due to a property of the image sensor, thesmoothing processing is often used for emphasizing a perspective.

SUMMARY

In the above technique in the past, it was difficult to emphasize theperspective to a higher degree by the smoothing processing. Theabove-described image pickup apparatus linearly varies a degree of thesmoothing processing with respect to a distance from a focusingposition. However, the perspective is more greatly emphasized when thedegree of the smoothing processing is varied non-linearly with respectto the distance from the focusing position. Suppose the degree (that is,blurring amount) of the smoothing processing is set on a distance S1from the focusing position and a distance S2 twice as long as thedistance S1, for example. In this case, when the blurring amount B1 of asubject in S1 is set larger than half the blurring amount B2 of asubject in S2, the perspective is more emphasized. Though such emphasisof the perspective can be performed by a user manually, complicatedoperations are necessary for the emphasis.

In view of the above problem, it is desirable to provide an imageprocessing device that emphasizes perspective of an image by performingimage processing.

According to embodiments of the present disclosure, there is provided animage processing device which includes an image acquisition partacquiring an image, a depth acquisition part acquiring a depthassociated with a pixel in the image, a depth conversion part convertingthe depth in accordance with a function having a characteristic tononlinearly approach a predetermined value with an increase in thedepth, and a storage part storing the converted depth in associationwith the image, and there are provided a method of controlling the imageprocessing device and a program for causing a computer to execute themethod. Accordingly, the depth converted in accordance with the functionhaving the characteristic to nonlinearly approach the predeterminedvalue with an increase in the depth is stored in association with theimage.

According to embodiments of the present disclosure, there may be furtherincluded a smoothing processing part performing smoothing processing onthe image to a degree depending on the converted depth corresponding tothe pixel in the image based on a converted depth corresponding to apredetermined pixel in the image. Accordingly, the smoothing processingdepending on the converted depth is performed on the image.

According to embodiments of the present disclosure, the function is thefunction with a characteristic that varies depending on a coefficient,and the smoothing processing part may perform the smoothing processingbased on the depth converted in accordance with the characteristic.Accordingly, the smoothing processing based on the depth converted inaccordance with the characteristic depending on the coefficient isperformed.

Further, according to embodiments of the present disclosure, thefunction may be an exponential function letting the depth be x, anoutput be y, a base of natural logarithm be e, a predetermined constantbe α and the coefficient be β, and defined as the following formulay=α×e{circumflex over ( )}(−βx). Accordingly, the depth is converted inaccordance with the function defined by the above-described formula.

Still further, according to embodiments of the present disclosure, thefunction may be an exponential function letting the depth be x, anoutput be y, a base of natural logarithm be e, a predetermined constantbe α and the coefficient be β, and defined by the following formulay=α×{1 e{circumflex over ( )}(−βx)}. Accordingly, the depth is convertedin accordance with the function defined by the above-described formula.

Still further, according to embodiments of the present disclosure, theremay be further included a coefficient supply part supplying a value ofthe coefficient depending on a shooting condition under which the imageis captured. Accordingly, the coefficient depending on the shootingcondition is supplied.

Still further, according to embodiments of the present disclosure, thestorage part may further store the shooting condition in associationwith the image, and the coefficient supply part may supply thecoefficient depending on the stored shooting condition. Accordingly, thecoefficient depending on the stored shooting condition is supplied.

Still further, according to embodiments of the present disclosure, thedepth conversion part may create an aggregation of the pixels as a depthimage, each pixel having the converted depth value as a pixel value.Accordingly, the aggregation of the pixels is created as the depth imagein which each pixel has the converted depth value as the pixel value.

Still further, according to embodiments of the present disclosure, theremay be further included a compression part compressing the depth imagein accordance with a predetermined image compression format, and thestorage part may store the compressed depth image in association withthe image. Accordingly, the depth image is compressed.

According to an embodiment of the present technology, the imageprocessing device is advantageous to emphasize a perspective in an imageby image processing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of animage pickup apparatus according to a first embodiment;

FIG. 2 is a block diagram illustrating a configuration example of animage processing device according to the first embodiment;

FIG. 3 is a graph illustrating an example of a relationship between agradation value and a subject distance according to the firstembodiment;

FIGS. 4A, 4B are diagrams illustrating examples of image data and depthimage data according to the first embodiment;

FIG. 5 is a diagram explanatory of a relationship between a focal pointdistance and the depth of field according to the first embodiment;

FIG. 6 is a diagram illustrating an example of a relationship of thedepth of field with the focal point distance, an aperture value, thesubject distance and a coefficient β according to the first embodiment;

FIG. 7 is a diagram illustrating an example of a relationship of thedepth of field with the focal point distance and the subject distanceaccording to the first embodiment;

FIG. 8 is a diagram illustrating an example of a data structure of adata file according to the first embodiment;

FIG. 9 is a flowchart illustrating an operation example of the imagepickup apparatus according to the first embodiment;

FIG. 10 is a flowchart illustrating an example of shooting processingaccording to the first embodiment;

FIG. 11 is a flowchart illustrating an example of smoothing processingaccording to the first embodiment;

FIG. 12 is a graph illustrating an example of an adjustable range of acoefficient according to the first embodiment;

FIGS. 13A, 13B are overall views illustrating a configuration example ofthe image pickup apparatus according to the first embodiment;

FIG. 14 is a graph illustrating an example of a relationship between agradation value and a subject distance according to a modificationaccording to the first embodiment;

FIG. 15 is a block diagram illustrating a configuration example of animage processing device according to a second embodiment;

FIG. 16 is a graph illustrating an example of a relationship between agradation value and a subject distance according to the secondembodiment in the case where a shooting mode is in a macro mode;

FIG. 17 is a graph illustrating an example of a relationship between thegradation value and the subject distance according to the secondembodiment in the case where the shooting mode is in a landscape mode;

FIG. 18 is a flowchart illustrating an example of shooting processingaccording to the second embodiment;

FIG. 19 is a flowchart illustrating an example of coefficient settingprocessing according to the second embodiment;

FIG. 20 is a block diagram illustrating a configuration example of animage processing device according to a third embodiment;

FIG. 21 is a diagram illustrating an example of a data structure ofattached information according to the third embodiment;

FIG. 22 is a flowchart illustrating an example of shooting processingaccording to the third embodiment; and

FIG. 23 is a flowchart illustrating an example of smoothing processingaccording to the third embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

The preferred embodiments (hereinafter referred to as embodiments)according to the present technology will be described below in thefollowing order.

1. First Embodiment (Image Processing: Example of Converting Depth Datato Depth Image Data)

2. Second Embodiment (Image Processing: Example of Changing CoefficientValue Based on Shooting Mode)

3. Third Embodiment (Image Processing: Example of Changing CoefficientValue after Storing Image Data)

1. First Embodiment [Image Pickup Apparatus Configuration Example]

FIG. 1 is a block diagram illustrating a configuration example of animage pickup apparatus 100 according to a first embodiment. The imagepickup apparatus 100 includes an operation part 110, a shooting lens130, an image sensor 140, an analog signal processing part 150, an A/D(Analog/Digital) conversion part 160, an image memory 170 and a workmemory 180. Further, the image pickup apparatus 100 includes an imagedata storage part 190, a display part 200 and an image processing device300. The image processing device 300 includes a camera control part 310and an image pickup apparatus control part 320.

The operation part 110 outputs an operation signal in response to a useroperation on a touch panel, a button or the like to the image processingdevice 300 via a signal line 111. The operation will be described belowin detail.

The shooting lens 130 is the lens for shooting an image. The imagesensor 140 converts light from the shooting lens 130 to an electricalsignal. The image sensor 140 outputs the converted electrical signal tothe analog signal processing part 150 via a signal line 141. The analogsignal processing part 150 performs predetermined analog signalprocessing on the electrical signal. The analog signal processingincludes CDS (Correlated Double Sampling) that cancels an amplifiernoise and a reset noise and AGC (automatic Gain Control) thatautomatically controls a gain. The analog signal processing part 150outputs the electrical signal after performing the processing to the A/Dconversion part 160 via a signal line 151.

The A/D conversion part 160 converts an analog electrical signal to adigital signal. The A/D conversion part 160 outputs the converteddigital signal to the image processing device 300 via a signal line 161as image data. Such image data is referred to as RAW image data becauseimage processing such as demosaic processing or compression processingis not performed on the image data at a time point when the image datais output from the A/D conversion part 160.

The image memory 170 temporarily holds the image data. The work memory180 temporarily holds the contents of work performed by the image pickupapparatus control part 320. The image data storage part 190 stores theimage data. The display part 200 displays an image based on the imagedata.

The camera control part 310 performs zoom control and exposure controlin accordance with control by the image pickup apparatus control part320 to acquire image data from the A/D conversion part 160. The cameracontrol part 310 acquires a depth related to a pixel in the image datafrom the acquired image data. The camera control part 310 converts thedepth in accordance with a predetermined function. The conversion methodwill be described below in detail. The camera control part 310 generatesan image obtained by aggregation of pixels each having the converteddepth as a gradation value. The generated image is an image(hereinafter, referred to as “depth image”) in which a depth of asubject in the image is represented by the gradation values of thepixels in a region of the subject. The camera control part 310 outputsdata of the depth image as depth image data to the image pickupapparatus control part 320 together with the image data.

The image pickup apparatus control part 320 controls the whole of theimage pickup apparatus 100. In particular, the image pickup apparatuscontrol part 320 performs zoom control and exposure control via thecamera control part 310 in response to the operation signal from theoperation part 110. The image pickup apparatus control part 320 receivesthe image data and the depth image data from the camera control part 310and stores the depth image data in the image data storage part 190 inassociation with the image data. Further, the image pickup apparatuscontrol part 320 reads out the image data from the image data storagepart 190 via a signal line 302 in response to the operation signal fromthe operation part 110. The image pickup apparatus control part 320performs image processing such as smoothing processing on the image databased on the corresponding depth image data. The image processing willbe described below in detail. As well as storing the image data afterthe image processing in the image data storage part 190, the imagepickup apparatus control part 320 outputs the image data to the displaypart 200 via a signal line 303 to cause the display part 200 to displaythe image data.

[Image Processing Device Configuration Example]

FIG. 2 is a block diagram illustrating a configuration example of theimage processing device 300 according to the first embodiment. Asdescribed above, the image processing device 300 includes the cameracontrol part 310 and the image pickup apparatus control part 320. Thecamera control part 310 includes a lens drive part 311, an imageacquisition part 312, a depth acquisition part 313, a depth conversionpart 314 and a smoothing processing part 315. The image pickup apparatuscontrol part 320 includes an operation signal analysis part 321, animage compression part 322, a depth image data addition part 323 and animage control part 324.

The operation signal analysis part 321 analyzes the operation signalfrom the operation part 110. Here, a user can change a zoom magnifyingpower, a degree of emphasis of the perspective in the smoothingprocessing and the like by operating the operation part 110. Theoperation signal analysis part 321 analyzes the operation signal, andwhen the zoom magnification power is changed, outputs the changed valueof the zoom magnification power to the lens drive part 311. Further,when the degree of emphasis of the perspective is changed, the operationsignal analysis part 321 outputs the changed degree of emphasis to theimage control part 324.

The lens drive part 311 controls a position of the shooting lens 130. Inparticular, the lens drive part 311 acquires a current position of theshooting lens 130 via a signal line 301 when receiving the changed valueof the zoom magnification power from the operation signal analysis part321. Subsequently, the lens drive part 311 outputs a control signal tocontrol, based on the changed value of the zoom magnification power, theposition of the shooting lens 130 to the shooting lens 130 via thesignal line 301.

The image acquisition part 312 acquires captured image data. Theacquired image data is temporarily held by the image memory 170. Theimage acquisition part 312 outputs the acquired image data to the imagecompression part 322.

The depth acquisition part 313 acquires a depth corresponding to a pixelin the captured image data. For example, the depth acquisition part 313detects a gap (phase difference) between two images of a subjectseparated by a separator lens and calculates a distance to the subjectas a depth based on the detected phase difference. The depth acquisitionpart 313 outputs the depth calculated in association with the pixel asdepth data to the depth conversion part 314. Note that, the depthacquisition part 313 may acquire the depth by a method other than thephase difference detection. For example, the depth acquisition part 313may irradiate laser beams on the subject and detects the reflected lightof the laser beams to measure a depth based on a delay time of thedetection time from the irradiation time.

The depth conversion part 314 converts the depth in accordance with apredetermined function. Such function is defined by the followingformula 1 and formula 2, for example. Note that, in formula 1, xrepresents a depth, y represents an output (i.e., converted depth) ofthe function, e represents base of natural logarithm and β represents acoefficient of a real number larger than 0. In the following formula 2,n represents an integer (e.g., “16”) not less than 1.

Y=αe{circumflex over ( )}(−βx)   Formula 1

α=2{circumflex over ( )}n−1   Formula 2

Note that, the depth conversion part 314 may convert the depth by usinga function other than the function defined by formula 1. It ispreferable that the function to be used has such a character that theoutput y non-linearly approaches a predetermined value (e.g., 0) inresponse to increase in the depth x. For example, the depth conversionpart 314 may use a logarithm function defined by a formula modified fromformula 1 or a function defined by formula 12 described below.

The depth conversion part 314 sets a predetermined initial value to thecoefficient β and converts the depth x to y to create image dataobtained by aggregation of pixels each having y as a gradation value.The depth conversion part 314 outputs the created image data to thedepth image data addition part 323 as the depth image data. Note that,the depth conversion part 314 may convert y to a pixel value furtherincluding a color phase (red, green, blue or the like) in addition tothe gradation value. For example, the depth conversion part 314 maychange the gradation value with respect to each color depending on ysuch that the smaller a value of y is, the closer to red the gradationvalue is, and the larger the value of y is, the closer to blue thegradation value is.

The image compression part 322 compresses the image data as necessary inaccordance with a predetermined image compression scheme. The imagecompression part 322 uses the work memory 180 as a work area during theimage compression processing. For example, JPEG (Joint PhotographicExperts Group) is used as the image compression scheme. The imagecompression part 322 outputs the compressed image data to the depthimage data addition part 323.

Note that, the image compression part 322 may output the image datauncompressed to the depth image data addition part 323. Alternatively,the image compression part 322 may further compress the depth imagedata. In this case, it is desirable that the compression technology usedfor the depth image data is the same as the compression technology usedfor the image data.

The depth image data addition part 323 stores the image data output fromthe image compression part 322 in association with the depth image dataoutput from the depth conversion part 314. The depth image data additionpart 323 associates the depth image data with the image data by addingthe depth image data to the image data as attached information (i.e.,tag) in Exif (Exchangeable image file format), for example. The depthimage data addition part 323 outputs the image data with the associateddepth image data to the image control part 324 as a data file. Notethat, the depth image data addition part 323 is an example of a storagepart according to embodiments of the present disclosure.

The image control part 324 manages the image data. In particular, theimage control part 324 stores the data file created by the depth imagedata addition part 323 in the image data storage part 190. Further, theimage control part 324 reads out a data file including the image data tobe displayed from the image data storage part 190 and outputs the readout image data to the display part 200 to cause the display part 200 todisplay the image data. Still further, the image control part 324 readsout, when a setting value of the emphasizing degree of the perspectiveis set by the operation signal analysis part 321, a data file includingimage data of a smoothing processing target from the image data storagepart 190 and outputs the data file to the smoothing processing part 315together with the setting value. Subsequently, the image control part324 receives smoothed image data from the smoothing processing part 315and outputs the smoothed image data to the image data storage part 190and the display part 200.

The smoothing processing part 315 performs smoothing processing on imagedata. In particular, when receiving the data file and the setting valuefrom the image control part 324, the smoothing processing part 315performs smoothing processing on the image data in the data file basedon the depth image data. However, the emphasizing degree of theperspective may be changed by a user through the operation part 110 asdescribed above. The smoothing processing part 315 updates, when theemphasizing degree of the perspective is changed, a value of thecoefficient β in formula 1 depending on the change. In particular, inthe case where the user changes the emphasizing degree of theperspective to be stronger, the value of the coefficient β is updated tobe larger, and in the case where the user changes the emphasizing degreeof the perspective to be weaker, the value of the coefficient β isupdated to be smaller. Then, the smoothing processing part 315 performssmoothing processing after updating the depth image data based on theupdated coefficient β. In the smoothing processing, the smoothingprocessing part 315 performs smoothing processing on the image data,based on a gradation value of a pixel determined as a focal point in thedepth image data, with a degree corresponding to a difference betweenthe gradation value of the pixel determined as the focal point and agradation value of each pixel. The smoothing processing part 315 outputsthe smoothed image data to the image control part 324.

FIG. 3 is a graph illustrating an example of a relationship between thegradation value and the subject distance according to the firstembodiment. In the figure, a horizontal axis represents the subjectdistance, that is, the depth, and a vertical axis represents thegradation value. A unit of the depth is measured in meter (m), forexample. A solid line represents a graph of a function defined byformula 1, and a dashed line represents a graph of a function defined byformula 3. Note that, in the following formula 3, x represents a depth,y represents an output of the function and a γ represents a coefficientof a real number larger than 0.

Y=(2{circumflex over ( )}16−1)−γx   Formula 3

In formula 3, the depth is converted to a gradation value linearlydecreasing in response to increase in depth.

Here, let a depth of a focused position in the smoothing processing bexf. And, let a depth deeper than xf be x1, and a depth further deeperthan x1 be x2. Let the gradation value y obtained by substituting thedepths xf, x1 and x2 in formula 1 be yf_e, y1_e and y2_e, respectively.Let the gradation value y obtained by substituting the depths xf, x1 andx2 in formula 3 be yf_L, y1_L and y2_L, respectively. And, let adifference between xf and x1 be Δx1 and a difference between xf and x2be Δx2. Further, let a difference between yf_e and y1_e be Δy1_e and adifference between yf_e and y2_e be Δy2_e. Still further, let adifference between yf_L and y1_L be Δy1_L and a difference between yf_Land y2_L be Δy2_L.

In the smoothing processing, the smoothing processing is performed,depending on a gradation value corresponding to a depth x_f of thefocused position, to a degree depending on a gradation valuecorresponding to each pixel in the image. For example, a blurring amountB representing a smoothing degree is calculated by the following formula4. Note that, in the following formula 4, A represents a coefficient ofa real number and Δy represents a difference between a gradation valueof the focused position and a gradation value of a position on whichsmoothing is to be performed. In particular, Δy1_e, Δy2_e, Δy1_L, Δy2_Lor the like is substituted into Δy.

B=A×Δy   Formula 4

The smoothing processing is performed by using a Gaussian filter definedby the following formula 5 through formula 7, for example. However, I(xp+k, yp+1) in the following formula 5 represents a pixel value of apixel on a coordinate (xp+k, yp+1) before performance of smoothingprocessing. In formula 5, r represents a radius of the Gaussian filterand an integer not less than 0 and w (k, 1) represents a weightcoefficient by which the pixel value I (xp+k, yp+1) is to be multiplied.Further in formula 6, σ represents a standard deviation and apredetermined real number is set. By the following formula 5 and formula6, the weight coefficient is set to be larger at a position closer tothe center of the Gaussian filter, and set to be smaller at a positioncloser to a surrounding area. In the following formula 7, “round( )” isa function returning an integer value not less than 0 by performingpredetermined rounding on a value shown in parentheses. For example,rounding half up or rounding off is performed as the rounding. Notethat, the smoothing processing part 315 may perform the smoothingprocessing by using a filter (e.g., mean filter) other than the Gaussianfilter.

$\begin{matrix}{{I^{\prime}( {x_{p},y_{p}} )} = {\sum\limits_{k = {- r}}^{r}{\sum\limits_{l = r}^{r}{{w( {k,l} )}{I( {{x_{p} + k},{y_{p} + 1}} )}}}}} & {{Formula}\mspace{14mu} 5} \\{{w( {x_{p},y_{p}} )} = {\frac{1}{2{\pi\sigma}^{2}}\exp \{ \frac{- ( {x_{p}^{2} + y_{p}^{2}} )}{2\sigma^{2}} \}}} & {{Formula}\mspace{14mu} 6} \\{R = {{round}\mspace{14mu} ( {B/2} )}} & {{Formula}\mspace{14mu} 7}\end{matrix}$

When formula 3 is used, smoothing processing is performed to a degree inproportion to difference in depth. For example, a ratio of Δy2_L toΔy1_L is equal to a ratio of Δx2 to Δx1. Accordingly, a ratio of ablurring amount B2_L of a subject in x2 to a blurring amount B1_L of asubject in x1 is equal to the ratio of Δx2 to Δx1.

On the other hand, when formula 1 is used, a blurring amount nonlinearlyrelated to the difference in depth is set. For example, a ratio of Δy2_Lto Δy1_L is larger than a ratio of Δx2 to Δx1. Accordingly, a ratio of ablurring amount B2_e of a subject in x2 to a blurring amount B1_e of asubject in x1 is larger than the ratio of Δx2 to Δx1. As a result,perspective is emphasized to a higher degree in comparison with the casewhere formula 3 is used. When a value of the coefficient β in formula 1is changed, a characteristic of formula 1 varies thereby to easilychange the emphasis degree of the perspective. In particular, the largerthe coefficient β is, the higher the emphasis degree of the perspectiveis, and the smaller the coefficient β is, the lower the emphasis degreeof the perspective is.

FIGS. 4A and 4B are diagrams illustrating examples of the image data andthe depth image data according to the first embodiment. FIG. 4Aillustrates an example of the image data 500. FIG. 4B illustrates anexample of the depth image data 510 created by using formula 1. Theimage data 500 shows a cube 501. The depth image data 510 is set atgradation values corresponding to acquired depths. For example, becausea depth of a vertex of a cube 511 corresponding to the cube 501 issmallest, a gradation value at the portion of the vertex is set at thelargest value to become bright. On the other hand, a gradation value ofa background having the largest depth is set at the smallest vale tobecome dark.

FIG. 5 is a diagram explanatory of a relationship between a focal pointdistance f and the depth of field DOF according to the first embodiment.The focal point distance f is a distance from the center of the shootinglens 130 to the focal point. The depth of field DOF is a range keeping asubject to be focused in an image even in the case where the subject ismoved along a depth direction. The focal point distance f and the depthof field DOF are measured in meters (m), for example. The shallower thedepth of field DOF is, the larger the blurring amount becomes in adefocused region. The depth of field DOF of a range on the near side ofthe subject is referred to as a near depth of field DN. On the otherhand, the depth of field DOF of a range on the far side of the subjectis referred to as a far depth of field DF.

Here, when a subject in a certain depth is focused, a depth withinfinity barely passing the farthest borderline of the depth of fieldDOF is referred to as a hyper focal point distance H. The hyper focalpoint distance H is represented as the following formula 8. Note that,in formula 8, N represents an aperture value, c represents a diameter ofa permissive circle of confusion in which blur in image is permissive.

H=f{circumflex over ( )}2/(Nc)   Formula 8

The depth of field DOF is calculated by the following formula 9 throughformula 11.

$\begin{matrix}{D_{N} = \frac{s( {H - f} )}{H + s - {2f}}} & {{Formula}\mspace{14mu} 9} \\{D_{F} = \frac{s( {H - f} )}{H - s}} & {{Formula}\mspace{14mu} 10} \\{{DOF} = {D_{F} + D_{N}}} & {{Formula}\mspace{14mu} 11}\end{matrix}$

FIG. 6 is a diagram illustrating an example of a relationship of thedepth of field DOF with the focal point distance f, the aperture valueN, the subject distance f and the coefficient β according to the firstembodiment. According to formula 8 through formula 11, the longer thefocal point distance f is, and the larger the aperture value N is, theshallower the depth of field DOF becomes, and the larger the blurringamount becomes. And the nearer the subject distance x is, the smallerthe hyper focal point distance H becomes and the shallower the depth offield DOF becomes according to formula 8 through formula 11. On theother hand, the larger the coefficient β is, the larger the blurringamount becomes according to formula 1. As described above, by changingthe value of the coefficient β, the blurring amount can be changedwithout changing the focal point distance f, the aperture value N andthe subject distance x.

FIG. 7 is a graph illustrating an example of a relationship of the depthof field DOF with the focal point distance f and the subject distance xbased on formula 5 through formula 7. Circular plots illustrate a casewhere the focal point distance f is 100 m. Quadrangular plots illustratea case where the focal point distance f is 80 m. Triangular plotsillustrate a case where the focal point distance f is 50 m. F number isfixed at 3.5. It is clear from FIG. 7 that the smaller the subjectdistance x is and the longer the focal point distance f is, theshallower the depth of field DOF becomes in the case where the F numberis fixed at a constant value.

[Data File Structure]

FIG. 8 is a diagram illustrating an example of a data structure of adata file according to the first embodiment. The data file is createdunder the Exif standards, for example.

In the data file, a start of image (SOI), an application marker segment1 (APP1), a define quantization table (DQT) and a define Huffman table(DHT) are sequentially stored. Then, following a start of frame header(SOF) and a start of scan header (SOS), a main body of compressed datais stored and an end of image (EOI) is stored. The compressed data isthe data compressed in accordance with a compression format such as JPEGstandards. Then, the depth image data created in the image processingdevice 300 is stored next to the end of image (EOI). Note that, thoughthe image processing device 300 stores the depth image data next to theEOI of the Exif standards, the depth image data may be stored as long asthe depth image data can be associated with the image data.

The APP1 is an area in which Exif attachment information is stored. Inthe APP1, an APP1 length is defined after an APP1 marker. Subsequently,after an Exif identifier code, a TIFF header, a principal image IFD (0thIFD), a principal image IFD value (0th IFD value) and the like arestored.

[Image Pickup Apparatus Operation Example]

FIG. 9 is a flowchart illustrating an operation example of the imagepickup apparatus 100 according to the first embodiment. This operationstarts when the image pickup apparatus 100 is powered on, for example.The image pickup apparatus 100 determines whether own current status isin a still image shooting mode (step S910). In the case where the owncurrent status is in the still image shooting mode (step S910: Yes), theimage pickup apparatus 100 performs shooting processing for shooting asubject (step S920).

In the case where the own current status is not in the still imageshooting mode (step S910: No) or after performing step S920, the imagepickup apparatus 100 determines whether the own current status is in astill image editing mode (step S930). In the case where the own currentstatus is in the still image editing mode (step S930: Yes), the imagepickup apparatus 100 performs smoothing processing (step S940). In thecase where the own current status is not in the still image editing mode(step S930: No) or after performing step S940, the image pickupapparatus 100 returns to step S910.

FIG. 10 is a flowchart illustrating an example of the shootingprocessing according to the first embodiment. The image pickup apparatus100 determines whether a shutter button is pressed (step S921). In thecase where the shutter button is pressed (step S921: Yes), the imageprocessing device 300 in the image pickup apparatus 100 acquires imagedata (step S922).

In the case where the shutter button is not pressed (step S921: No) orafter performing step S922, the image processing device 300 creates thedepth data based on the image data (step S923). Then, the imageprocessing device 300 creates the depth image data from the depth databy using formula 1 (step S924). The image processing device 300compresses the image data as necessary (step S925). The image processingdevice 300 stores the image data by adding the depth image data to theimage data (step S926). After performing step S926, the image pickupapparatus 100 terminates the shooting processing.

FIG. 11 is a flowchart illustrating an example of the smoothingprocessing according to the first embodiment. The image pickup apparatus100 accepts processing for selecting image data on which the smoothingprocessing is to be performed (step S941). The image pickup apparatus100 determines whether the image data is selected (step S942). In thecase where the image data is not selected (step S942: No), the imagepickup apparatus 100 returns to step S942. In the case where the imagedata is selected (step S942: Yes), the image pickup apparatus 100accepts an operation setting the degree of the smoothing processing asthe emphasis degree of the perspective (step S943). Subsequently, theimage pickup apparatus 100 determines whether the degree is set (stepS944).

In the case where the degree is set (step S944: Yes), the imageprocessing device 300 in the image pickup apparatus 100 changes thecoefficient β depending on the set value of the degree (step S945). Theimage processing device 300 updates the depth image data based on thechanged coefficient β (step S946). The image processing device 300performs the smoothing processing on the image data based on the updateddepth image data (step S947). The image pickup apparatus 100 displaysthe image data after the smoothing processing (step S948).

In the case where the degree is not set (step S944: No) or afterperforming step S948, the image pickup apparatus 100 determines whetheran exit operation of edition is performed (step S949). In the case wherethe exit operation of edition is not performed (step S949: No), theimage pickup apparatus 100 returns to step S944. In the case where theexit operation of edition is performed (step S949: Yes), the imagepickup apparatus 100 stores the image data after the soothing processing(step S950). After performing step S950, the image pickup apparatus 100terminates the smoothing processing.

FIG. 12 is a graph illustrating an example of an adjustable range of thecoefficient β according to the first embodiment. When a value of thecoefficient β is changed within a certain range, a functioncharacteristic of formula 1 varies. The larger the coefficient β is, thecloser a curved line of the function in formula 1 approaches a straightline where y=0. As a result, the emphasis degree of the perspective isincreased. On the contrary, the lower the coefficient β is, the more theemphasis degree of the perspective is decreased.

FIGS. 13A, 13B are overall views illustrating a configuration example ofthe image pickup apparatus 100 according to the first embodiment. FIG.13A illustrates an example of a top face and a front face of the imagepickup apparatus 100 and FIG. 13B illustrates an example of a back faceof the image pickup apparatus 100. On the top face of the image pickupapparatus 100, a zoom lever 101, a shutter button 102, a play button 103and a power button 104 are provided. On the front face of the imagepickup apparatus 100, a shooting lens 105, an AF (Auto Focus)illuminator 106 and a lens cover 107 are provided. On the back face ofthe image pickup apparatus 100, a touch screen 108 is provided.

The zoom lever 101 is a button for performing a zoom control operation.The shutter button 102 is a button for shooting photos of a subject. Theplay button 103 is a button for displaying image data. The power button104 is a button for powering on or off the image pickup apparatus 100.The shooting lens 105 is the lens for capturing an image. The AFilluminator 106 emits light when an autofocus function is activated. Thelens cover 107 is a component movable to a position to cover the lensfor protecting the lens. The touch screen 108 is a display enablingoperations of the image pickup apparatus 100 by touch of a finger or thelike.

The operation part 110 illustrated in FIG. 1 includes the zoom lever101, the shutter button 102, the play button 103 and the power button104 illustrated in FIG. 13A. The operation part 110 and the display part200 illustrated in FIG. 1 includes the touch screen 108 illustrated inFIG. 13B.

As described above, according to the first embodiment of the presenttechnology, the image processing device 300 acquires the image and thedepth and converts the depth in accordance with the function having thecharacteristic to nonlinearly approach the predetermined value withincrease in depth. The image processing device 300 stores the converteddepth in association with the image. When the image processing device300 performs the smoothing processing based on the converted depth, theperspective is emphasized to a degree higher than the degree in thesmoothing processing proportional to the depth.

[First Modification]

A first modification of the first embodiment will be described withreference to FIG. 14. Unlike the first embodiment, an image processingdevice 300 of the first modification converts the depth such that agradation value y increases with increase in depth x. The imageprocessing device 300 according to the first modification converts thedepth by using the following formula 12, for example, instead of formula1.

Y=α{1−e{circumflex over ( )}(−βx)}  Formula 12

FIG. 14 is a graph illustrating an example of a relationship between agradation value y and a subject distance (depth) x according to thefirst modification. In the first embodiment, the depth is converted tothe gradation value y that decreases nonlinearly depending on increasein depth x by using formula 1. On the other hand, in the case of usingformula 12 as illustrated in FIG. 14, the depth x is converted to thegradation value y that increases nonlinearly depending on increase indepth x.

2. Second Embodiment [Image Processing Device Configuration Example]

Next, a second embodiment of the present technology will be describedwith reference to FIG. 15 through FIG. 19. FIG. 15 is a block diagramillustrating a configuration example of an image processing device 300according to the second embodiment. As described above, the imageprocessing device 300 includes the camera control part 310 and the imagepickup apparatus control part 320. Unlike the first embodiment, theimage processing device 300 according to the second embodiment furtherincludes a coefficient supply part 316 in a camera control part 310.Furthermore, an operation signal analysis part 321 according to thesecond embodiment outputs a shooting mode among shooting conditionsfurther to the coefficient supply part 316. The shooting mode is theinformation indicating the shooting conditions such as the type of ashooting target and a distance to the shooting target. For example, theshooting mode includes a macro mode, a landscape mode and a normal mode.The macro mode is the mode for shooting a subject near the lens. Thelandscape mode is the mode for shooting a distant subject. The normalmode is the mode for shooting a subject at a distance between that inthe macro mode and that in the landscape mode.

The coefficient supply part 316 supplies the coefficient β depending onthe shooting conditions. In the coefficient supply part, values of thecoefficient β of the respective shooting mode are preliminarily set. Thecoefficient supply part 316 receives the shooting mode from theoperation signal analysis part 321 and outputs the coefficient βcorresponding to the received shooting mode to the depth conversion part314. For example, the value larger than the set value in the normal modeis set in the macro mode, and the value smaller than the set value inthe normal mode is set in the landscape mode. The depth conversion part314 substitutes the value of the coefficient β from the coefficientsupply part 316 in formula 12 to convert the depth.

FIG. 16 is a graph illustrating an example of a relationship between thegradation value and the subject distance according to the secondembodiment in the case where the shooting mode is in the macro mode. Asdescribed above, the value larger than the set value in the normal modeis set to the coefficient β in the macro mode. Accordingly, the emphasisdegree of the perspective becomes relatively high. For example, a ratioof Δy2_e to Δy1_e becomes larger than that in the normal mode, andaccordingly, the blurring amount of the subject at x1 becomes relativelylarge.

FIG. 17 is a graph illustrating an example of a relationship between thegradation value and the subject distance according to the secondembodiment in the case where the shooting mode is in the landscape mode.As described above, the value smaller than the set value in the normalmode is set to the coefficient β in the landscape mode. Accordingly, theemphasis degree of the perspective becomes relatively low. For example,a ratio of Δy2_e to Δy1_e becomes smaller than that in the normal mode,and accordingly, the blurring amount of the subject at x1 becomesrelatively small.

[Image Pickup Apparatus Operation Example]

FIG. 18 is a flowchart illustrating an example of the shootingprocessing according to the second embodiment. Unlike the firstembodiment, coefficient setting processing (step S960) is furtherperformed in the shooting processing according to the second embodimentafter performing step S923. The image pickup apparatus 100 performs stepS924 after performing step S960.

FIG. 19 is a flowchart illustrating an example of the coefficientsetting processing according to the second embodiment. The imageprocessing device 300 sets the initial value to the coefficient β. Here,the initial value is the setting value in the normal mode, for example(step S961). The image processing device 300 determines whether theshooting mode is the macro mode (step S962). In the case where theshooting mode is not the macro mode (step S962: No), the imageprocessing device 300 determines whether the shooting mode is thelandscape mode (step S963). In the case where the shooting mode is thelandscape mode (step S963: Yes), the image processing device 300 changesthe value of the coefficient β to a value smaller than the initial value(step S964). In the case where the shooting mode is the macro mode (stepS962: Yes), the image processing device 300 changes the value of thecoefficient β to a value larger than the initial value (step S965).

In the case where the shooting mode is not the landscape mode (stepS963: No) or after performing step S964 or step S965, the imageprocessing device 300 terminates the coefficient setting processing.

As described above, according to the second embodiment of the presenttechnology, the coefficient supply part 316 supplies the coefficient βdepending on the shooting conditions, and the depth conversion part 314converts the depth based on the supplied coefficient β. Because thecharacteristic of the function varies depending on the coefficient β,the depth is converted by the function of the characteristic dependingon the shooting conditions. As a result, the smoothing processingsuitable for the shooting conditions is performed based on the converteddepth.

3. Third Embodiment [Image Processing Device Configuration Example]

Next, a third embodiment of the present technology will be describedwith reference to FIG. 20 through FIG. 23. FIG. 20 is a block diagramillustrating a configuration example of an image processing device 300according to the third embodiment. As described above, the imageprocessing device 300 includes the camera control part 310 and the imagepickup apparatus control part 320. Unlike the first embodiment, in theimage processing device 300 according to the third embodiment, acoefficient supply part 316 supplies a coefficient depending on theshooting mode after the image data is stored. Further, an operationsignal analysis part 321 according to the third embodiment outputs theshooting mode further to a depth image data addition part 323. The depthimage data addition part 323 according to the third embodiment storesthe image data in the image data storage part 190 by adding the shootingmode to the image data. The image control part 324 outputs the shootingmode added to the image data to a coefficient supply part 316 whenreading out the image data.

The coefficient supply part 316 according to the third embodimentreceives the shooting mode from the image control part 324 and outputsthe coefficient β depending on the shooting mode to the smoothingprocessing part 315.

[Data File Structure]

FIG. 21 is a diagram illustrating an example of a data structure ofattached information in a data file according to the third embodiment.The attached information (tag) of the image data is stored in the 0thIFD in the application marker segment (APP1). The attached informationis segmented into segments such as a version tag, a user information tagand a shooting condition tag. The version tag includes an Exif versionand a corresponding flash pix version. The user information tag includesa maker note, a user comment and the like. The shooting condition tagincludes an exposure time, F-number, a shooting scene type, a subjectdistance range and the like. Here, whether a shooting target is normal,a landscape or a person is stored in an area of the shooting scene type.Further, it is stored in an area of the subject distance range which adistance to the subject is classified into macro, a near view or adistant view.

In the third embodiment, the normal mode, the landscape mode or themacro mode is set as the shooting mode. Information related to theshooting modes is stored in areas of the shooting scene type and thesubject distance range. Note that, the image processing device 300 maystore the information related to the shooting mode in another area suchas an area of the maker note.

[Image Pickup Apparatus Operation Example]

FIG. 22 is a flowchart illustrating an example of the shootingprocessing according to the third embodiment. Unlike the firstembodiment, the shooting processing according to the third embodiment,the image processing device 300 adds the shooting mode to the image dataafter performing step S926 (step S927). The image pickup apparatus 100terminates the shooting processing after performing step S927.

FIG. 23 is a flowchart illustrating an example of the smoothingprocessing according to the third embodiment. Unlike the firstembodiment, the image processing device 300 further performs coefficientsetting processing (step S960) in the case where the image data isselected (step S942: Yes) in the smoothing processing according to thethird embodiment. The coefficient setting processing (step S960) isprocessing similar to the coefficient setting processing according tothe second embodiment. The image processing device 300 accepts anoperation of setting a degree of the smoothing processing (step S943)after performing step S960.

As described above, according to the third embodiment, the depth imagedata addition part 323 stores the image data by adding the shootingcondition to the image data and the coefficient supply part 316 readsout the shooting conditions and supplies the coefficient depending onthe shooting conditions. Because the characteristic of the functionvaries depending on the coefficient β, the depth is converted by thefunction of the characteristic depending on the shooting conditions. Asa result, the smoothing processing suitable for the shooting conditionsis performed based on the converted depth.

The above-described embodiments indicate examples for embodying thepresent disclosure and matters according to the embodiments each havecorrespondence relation with claimed elements in the appended claims asexplained below. Similarly, claimed elements in the appended claims eachhave corresponding relation with matters according to the embodiments ofthe present disclosure having the same name. However, the presentdisclosure is not limited to the embodiments. Various modifications canbe applied to the present disclosure without departing from the spiritof the present disclosure.

Further, the above-described procedures in the above embodiments may beregarded as a method having the series of steps or as a program causinga computer to execute the series of steps and as a storage mediumstoring the program. The storage medium may include CD (Compact Disc),MD (MiniDisc), DVD (Digital Versatile Disk), a memory card, a Blu-rayDisc (registered trademark), a nonvolatile memory such as a flash memoryand the like.

Additionally, the present technology may also be configured as below.

-   (1) An image processing device comprising:

an image acquisition part acquiring an image;

a depth acquisition part acquiring a depth associated with a pixel inthe image;

a depth conversion part converting the depth in accordance with afunction having a characteristic to nonlinearly approach a predeterminedvalue with an increase in the depth; and

a storage part storing the converted depth in association with theimage.

-   (2) The image processing device according to (1), further comprising    a smoothing processing part performing smoothing processing on the    image to a degree depending on the converted depth corresponding to    the pixel in the image based on a converted depth corresponding to a    predetermined pixel in the image.-   (3) The image processing device according to (2), wherein the    function is a function with a characteristic that varies depending    on a coefficient, and

the smoothing processing part performs the smoothing processing based onthe depth converted in accordance with the characteristic.

-   (4) The image processing device according to (3), wherein the    function is an exponential function defined by the following    formula:

Y=α×e{circumflex over ( )}(−βx), where x is the depth, y is an output, eis a base of natural logarithm, α is a predetermined constant and β isthe coefficient.

-   (5) The image processing device according to (3), wherein the    function is an exponential function defined by the following    formula:

Y=α×{1−e{circumflex over ( )}(−βx)}, where x is the depth, y is anoutput, e is a base of natural logarithm, α is a predetermined constantand β is the coefficient.

-   (6) The image processing device according to any one of (3) to (5),    further comprising a coefficient supply part supplying a value of    the coefficient depending on a shooting condition under which the    image is captured.-   (7) The image processing device according to (6), wherein the    storage part further stores the shooting condition in association    with the image, and

the coefficient supply part supplies the coefficient depending on thestored shooting condition.

-   (8) The image processing device according to any one of (1) to (7),    wherein the depth conversion part creates an aggregation of the    pixels as a depth image, each pixel having the converted depth value    as a pixel value.-   (9) The image processing device according to (8), further comprising    a compression part compressing the depth image in accordance with a    predetermined image compression format, and

the storage part stores the compressed depth image in association withthe image.

-   (10) A method of controlling an image processing device comprising:

acquiring, with an image acquisition part, an image;

acquiring, with a depth acquisition part, a depth associated with apixel in the image;

converting, with a depth conversion part, the depth in accordance with afunction having a characteristic to nonlinearly approach a predeterminedvalue with an increase in the depth; and

storing, with a storage part, the converted depth in association withthe image.

-   (11) A program for causing a computer to execute:

acquiring an image;

acquiring a depth associated with a pixel in the image;

converting the depth in accordance with a function having acharacteristic to nonlinearly approach a predetermined value with anincrease in the depth; and

storing the converted depth in association with the image.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2011-182518 filed in theJapan Patent Office on Aug. 24, 2011, the entire content of which ishereby incorporated by reference.

1. (canceled)
 2. A device comprising: a memory; and circuitry, coupledwith the memory, configured to set an initial value of a coefficient ofa function used to convert depth associated with a pixel within imagedata to have a nonlinear characteristic with an increase in the depth,modify the initial value of the coefficient to a second value based on ashooting mode, and convert the depth associated with the pixel withinthe image data to have the nonlinear characteristic with the increase inthe depth, based the initial value or the second value, the nonlinearcharacteristic being different between the plurality of shooting modes,and different shooting modes having different converted depths.
 3. Thedevice according to claim 2, wherein the circuitry is configured tomodify the initial value of the coefficient to the second value that issmaller than the initial value, after setting the initial value of thecoefficient and in response to the shooting mode being a first shootingmode.
 4. The device according to claim 2, wherein the circuitry isconfigured to modify the initial value of the coefficient to the secondvalue that is greater than the initial value, after setting the initialvalue of the coefficient and in response to the shooting mode being asecond shooting mode.
 5. The device according to claim 2, wherein thecircuitry is further configured to perform smoothing processing on theimage data to a degree depending on the converted depth corresponding tothe pixel in the image data based on a converted depth corresponding toa predetermined pixel in the image data.
 6. The device according toclaim 5, wherein the circuitry converts the depth associated with thepixel within the image data in accordance with the function tononlinearly approach a predetermined value with the increase in thedepth, the function being a function with a characteristic that variesdepending on the coefficient, and performs the smoothing processingbased on the depth converted in accordance with the characteristic. 7.The device according to claim 2, wherein the function is an exponentialfunction defined by the following formula: Y=α×e{circumflex over( )}(−βx), where x is the depth, y is an output, e is a base of naturallogarithm, α is a predetermined constant and β is the coefficient. 8.The device according to claim 2, wherein the function is an exponentialfunction defined by the following formula: Y=α×{1−e{circumflex over( )}(−βx)}, where x is the depth, y is an output, e is a base of naturallogarithm, α is a predetermined constant and β is the coefficient. 9.The device according to claim 2, wherein the circuitry is furtherconfigured to create an aggregation of pixels as a depth image, eachpixel having the converted depth value as a pixel value.
 10. The deviceaccording to claim 9, wherein the circuitry is further configured tocompress the depth image in accordance with a predetermined imagecompression format, and the circuitry is configured to store thecompressed depth image in association with the image data.
 11. A methodcomprising: setting an initial value of a coefficient of a function usedto convert depth associated with a pixel within image data to have anonlinear characteristic with an increase in the depth; modifying theinitial value of the coefficient to a second value based on a shootingmode; and converting the depth associated with the pixel within theimage data to have the nonlinear characteristic with the increase in thedepth, based on the initial value or the second value, the nonlinearcharacteristic being different between the plurality of shooting modes,and different shooting modes having different converted depths.
 12. Themethod according to claim 11, wherein the modifying modifies the initialvalue of the coefficient to the second value that is smaller than theinitial value, after setting the initial value of the coefficient and inresponse to the shooting mode being a first shooting mode.
 13. Themethod according to claim 11, wherein the modifying modifies the initialvalue of the coefficient to the second value that is greater than theinitial value, after setting the initial value of the coefficient and inresponse to the shooting mode being a second shooting mode.
 14. Themethod according to claim 11, further comprising performing smoothingprocessing on the image data to a degree depending on the converteddepth corresponding to the pixel in the image data based on a converteddepth corresponding to a predetermined pixel in the image data.
 15. Themethod according to claim 14, wherein the converting converts the depthassociated with the pixel within the image data in accordance with thefunction to nonlinearly approach a predetermined value with the increasein the depth, the function being a function with a characteristic thatvaries depending on the coefficient, and the performing performs thesmoothing processing based on the depth converted in accordance with thecharacteristic.
 16. A non-transitory computer-readable storage mediumincluding computer executable instructions, wherein the instructions,when executed by a computer, cause the computer to perform a method, themethod comprising: setting an initial value of a coefficient of afunction used to convert depth associated with a pixel within image datato have a nonlinear characteristic with an increase in the depth;modifying the initial value of the coefficient to a second value basedon a shooting mode; and converting the depth associated with the pixelwithin the image data to have the nonlinear characteristic with theincrease in the depth, based on the initial value or the second value,the nonlinear characteristic being different between the plurality ofshooting modes, and different shooting modes having different converteddepths.
 17. The non-transitory computer-readable storage mediumaccording to claim 16, wherein the modifying modifies the initial valueof the coefficient to the second value that is smaller than the initialvalue, after setting the initial value of the coefficient and inresponse to the shooting mode being a first shooting mode.
 18. Thenon-transitory computer-readable storage medium according to claim 16,wherein the modifying modifies the initial value of the coefficient tothe second value that is greater than the initial value, after settingthe initial value of the coefficient and in response to the shootingmode being a second shooting mode.
 19. The non-transitorycomputer-readable storage medium according to claim 16, furthercomprising performing smoothing processing on the image data to a degreedepending on the converted depth corresponding to the pixel in the imagedata based on a converted depth corresponding to a predetermined pixelin the image data.
 20. The non-transitory computer-readable storagemedium according to claim 19, wherein the converting converts the depthassociated with the pixel within the image data in accordance with thefunction to nonlinearly approach a predetermined value with the increasein the depth, the function being a function with a characteristic thatvaries depending on the coefficient, and the performing performs thesmoothing processing based on the depth converted in accordance with thecharacteristic.